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This  report  was  written  in  response  to  a  request  from  Under  Secretary  of  the  Army 
James  R.  Ambrose  for  a  review  of  the  report,  "Training  Matrix  Validation  and 
Verification  Test  Report  for  the  Ml  Unit-Conduct  of  Fire  Trainers  (U-COFT)."  The 
validation  and  verification  (V/V)  had  two  principal  thru.sis: 
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proficiency  on  the  U-COFT. 

The  reviewers  concluded  that: 

|(l)  Deficiency  identification  was  conducted  in  a  logical,  systematic  manner;  and 
resulted  in  useful  hardware  and  software  improvements^ 

2^  Thirty  hours  of  U-COFT  sustainment  training  produced  substantial  average 
gains  in  the  proficiency  of  a  test  group  comprising  five  TC-gunner  pairs^ 

3,  Forty  hours  of  U-COFT  transition  training  with  another  five  TC-gunner  pairs 
produced  average  exit-level  scores  equal  to  or  greater  than  those  obtained 
by  the  sustainment  pairs.  Proficiency  gains  for  the  transition  group,  while 
probably  substantial,  could  not  be  estimated  accurately,  because  entry-level 
proficiency  scores  were  not  obtained^, 

1 Proficiency  gains  in  target-acquisition  time  on  the  U-COFT  were  underesti¬ 
mated  by  at  least  one-third  in  the  V/V  report^ 

5^  Substantial  gains  were  found  in  percents  of  targets  acquired,  engaged,  hit, 
and  killed  by  the  sustainment  and  transition  groups.  The  gains  resulted 
from  improvements  in  acquisition  time,  engagement  time,  and  first  round  hits, 
which  in  turn  allowed  time  to  scan,  acquire,  and  engage  available  second  and 
third  targets,  6  \  X 

6}  Nine  hours  of  U-COFT  training  and  testing  produced  statistically  significant 
improvements  between  pretest  and  posttest  scores  of  eight  Canadian  Armor 
Trophy  (CAT)  TC-gunner  pairs  on  several  measures  of  gunnery  prof iciency. 
Statistically  significant  differences  were  not  found,  however,  in  percent 
main  gun  hits  or  in  target  coverage  with  the  coaxial  machinegun.  The  CAT 
teams  were  so  proficient  at  hitting  targets  before  U-COFT  training  began 
that  there  was  little  room  for  improvement. 

7.  Review  of  VU-graphs,  which  were  used  to  summarize  proficiency  before  and 
after  training  in  a  briefing  to  LTG  W.  F.  Ulmer,  revealed  that  the  reported 
performance  levels  and  gains  were  inconsistent  with  V/V  test  data.  The 
briefing  data  were  based,  not  on  pretest  and  posttest  scores,  but  on  V/V 
test  administrators'  impressions  of  soldiers'  progress  in  training. 

8.  The  V/V  findings  regarding  proficiency  retention  were  not  reliable  enough  to 
be  used  in  making  decisions  about  retention  or  retraining  intervals. 

9.  Consideration  should  be  given  to  expanding  the  content  of  TC  training  to  in¬ 
clude  more  prefiring  activities;  target  acquisition,  for  example,  and  making 
initial  main  gun  lays  for  gunners'  engagements. 

10.  The  TC  and  the  TC-gunner  tests  used  in  V/V  did  not  provide  adequate  sampling 
of  training  content.  NBC  gunnery,  for  example,  and  several  aspects  of 
degraded-mode  gunnery  were  practiced  but  not  tested.  Domain-referenced 
methods  should  be  used  for  constructing  U-COFT  and  other  tank  gunnery  tests 
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EXECUTIVE  SUMMARY 


REQUIREMENT 

The  requirement  addressed  by  this  report  was  Initiated  by  a  letter 
(11  March  1985)  from  the  Honorable  James  R.  Ambrose  to  the  USAMC  Commanding 
General,  and  others,  in  which  the  Under  Secretary  requested  that  "analysts 
who  are  not  partisans  in  the  U-COFT  program"  review  the  report,  "Training 
Matrix  Validation  and  Verification  Test  Report  for  the  Ml  Unit-Conduct  of 
Fire  Trainers  (U-COFT)."  Colonel  James  M.  Ball,  PM  TRADE,  asked  the  Field 
Unit  Chief,  US  Army  Research  Institute  (ARI)  at  Fort  Knox,  to  perform  the 
review.  This  report  is  the  response  to  that  request. 

APPROACH 

The  validation  and  verification  ("V/V")  had  two  principal  thrusts: 

1.  Identifying  and  correcting  deficiencies  in  U-COFT  hardware  and  software. 

2.  Ascertaining  the  extent  to  which  practice  with  the  U-COFT  Improved 
proficiency  on  the  U-COFT. 

The  V/V  report  was  examined  by  the  authors  of  this  review.  Our  review  was 
supplemented  by  information  from  other  sources,  including  briefings  and  addi¬ 
tional  analyses  of  data  provided  by  the  U-COFT  contractor,  General  Electric 
Company. 

CONCLUSIONS 

\  The  reviewers  concluded  that: 

1.  Deficiency  identification  was  conducted  in  a  logical,  systematic  manner; 
and  resulted  in  useful  hardware  and  software  improvements. 

2.  Thirty  hours  of  U-COFT  sustainment  training  produced  substantial  average 
gains  in  the  proficiency  of  a  test  group  comprising  five  TC-gunner  pairs. 

3.  Forty  hours  of  U-COFT  transition  training  with  another  five  TC-gunner 
pairs  produced  average  exit-level  scores  equal  to  or  greater  than  those 
obtained  by  the  sustainment  pairs.  Proficiency  gains  for  the  transition 
group,  while  probably  substantial,  could  not  be  estimated  accurately, 
because  entry-level  proficiency  scores  were  not  obtained. 

**•  Additional  analyses  of  test  data  suggested  that  proficiency  gains  in  tar¬ 
get-acquisition  time  on  the  U^OFT  were  underestimated . by  at  least  one- 
third  in  the  V/V  report. 

5.  Additional  analyses  of  test  data  also  showed  substantial  gains  in  percents 
of  targets  acquired,  engaged,  hit,  and  killed.  The  gains  resulted  from 
Improvements  in  acquisition  time,  engagement  time,  and  first  round  hits, 
which  in  turn  allowed  time  to  scan,  acquire,  and  engage  available  second 
and  third  targets. 
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6.  Nine  hours  of  U-COFT  training  and  testing  produced  statistically  signifi¬ 
cant  improvements  between  pretest  and  posttest  scores  of  eight  Canadian 
Armor  Trophy  (CAT)  TC-gunner  pairs  on  acquisition  time,  first  round  fire 
time,  first  target  hit  time,  second  target  fire  time,  second  target  hit 
time,  and  number  of  coaxial  machinegun  rounds  fired.  Statistically  sig¬ 
nificant  differences  were  not  found  in  percent  main  gun  hits  or  in  target 
coverage  with  the  coaxial  machinegun.  The  CAT  teams  were  so  proficient  at 
hitting  targets  before  U-COFT  training  began  that  there  was  little  room 
for  improvement. 

7.  Review  of  VU-graphs,  which  were  used  to  summarize  proficiency  before  and 
after  training  in  a  briefing  to  LTG  W.  F.  Ulmer,  revealed  that  the  re¬ 
ported  performance  levels  and  gains  were  inconsistent  with  V/V  test  data. 
The  briefing  data  were  based,  not  on  pretest  and  posttest  scores,  but  on 
V/V  test  administrators'  impressions  of  soldiers'  progress  in  training. 

8.  Proficiency  loss  was,  according  to  the  V/V  report,  "definite"  after  10 
weeks,  and  nonexistent  after  thi<.e.  These  findings  are  not  reliable 
enough  to  be  used  for  making  decisions  about  retention  or  retraining  in¬ 
tervals:  The  number  of  compared  TC-gunner  pairs  was  small  (two  each  in 
the  10-  and  3-week  groups),  and  the  transition  group  received  more  train¬ 
ing  than  did  the  sustainment  group. 

9.  Tank  Commander  training  exercises  dealt  mainly  with  firing.  Consideration 
should  be  given  to  expanding  the  content  of  TC  training  to  include  more 
prefiring  activities;  target  acquisition,  for  example,  and  making  initial 
main  gun  lays  for  gunners'  engagements. 

10.  The  TC  and  the  TC-gunner  tests  used  in  V/V  did  not  provide  adequate  sam¬ 
pling  of  training  content.  NBC  gunnery,  for  example,  and  several  aspects 
of  degraded-mode  gunnery  were  practiced  but  not  tested.  Domain- 
referenced  methods,  described  in  several  ARI  reports,  should  be  used  for 
constructing  U-COFT  and  other  tank  gunnery  tests. 
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REVIEW  OF  THE  Ml  UNIT-CONDUCT  OF  FIRE  TRAINER  (U-COFT) 

VALIDATION  AND  VERIFICATION  TEST  REPORT _ 

The  Honorable  James  R.  Ambrose,  in  a  letter  (11  March  1985)  to  the  USAMC 
Commanding  General  and  others,  requested  that  "analysts  who  are  not  partisans 
in  the  U-COFT  program"  review  the  report,  "Training  Matrix  Validation  and 
Verification  Test  Report  for  the  Ml  Unit-Conduct  of  Fire  Trainers  (U-COFT)" 
(General  Electric  Company,  9  March  198*0.  The  USAMC  Commanding  General  rec¬ 
ommended  to  the  USA  TRADOC  Commanding  General  that  the  US  Army  Research  In¬ 
stitute  "be  tasked  as  the  'non-partisan*  analyst"  mentioned  in  the  Under 
Secretary's  letter.  Colonel  James  M.  Ball,  PM  TRADE,  requested  (letter 
dated  16  May  in  Appendix  A)  that  ARI  Fort  Knox  examine  the  G.E.  report  and 
respond  to  the  Under  Secretary's  request.  This  report  summarizes  the  results 
of  the  requested  examination. 

INFORMATION  SOURCES 

The  review  of  the  G.E.  validation  and  verification  (V/V)  report  was 
supplemented  by  information  from  other  sources,  including: 

1.  Briefing  slides  from  G.E.  briefing,  "Ml  U-COFT  Instructional  Subsys¬ 
tem  Test  Results,"  to  LTG  Walter  F.  Ulmer,  16  January  1984. 

2.  G.E.  briefing  to  authors  of  this  report  on  14  May  1985. 

3.  Additional  analyses  of  data  summarized  in  the  G.E.  report. 

4.  U-COFT  data  (not  included  in  the  V/V  report)  from  eight  Canadian 
Armor  Trophy  (CAT)  crews. 

The  V/V  report  had  two  principal  thrusts: 

1.  Identifying  and  correcting  deficiencies  in  U-COFT  hardware  and  soft¬ 
ware. 

2.  Ascertaining  the  extent  to  which  practice  with  the  U-COFT  improved 
proficiency  on  the  U-COFT. 

DEFICIENCY  CORRECTIONS 

The  changes  made  as  a  result  of  G.E.'s  validation  and  verification  are 
documented  in  the  V/V  report,  and  were  described  in  the  14  May  briefing  by 
G.E.  A  sample  of  the  changes  is: 

1.  Rules  for  progressing  through  U-COFT  exercises  were  revised  to  pro¬ 
vide  practice  on  a  greater  number  and  variety  of  exercises,  and  to 
minimize  the  occurrence  of  situations  in  which  a  student  "bounces 
back  and  forth"  between  an  unsuccessfully  completed  exercise  and  a 
successfully  completed  one;  that  is,  situations  in  which  a  student 
falls  a  test  exercise,  is  directed  back  to  a  previously  passed  exer¬ 
cise,  passes  it  again,  fails  the  test  exercise,  and  so  forth. 


2.  Software  was  revised  to  improve  hardening  of  terrain  features  and 
other  "non-targets";  this  was  done  to  prevent  hitting  targets  by 
shooting  through  other  objects,  a'  i  to  prevent  students'  being 
"killed"  by  targets  shooting  thr'  gh  terrain  features. 

3.  Software  was  revised  to  provide  clearer  kill  indicators;  tank  tar¬ 
gets,  for  example,  now  tilt  forward  when  killed.  (Notice  that  en¬ 
hanced  kill  indicators  may  produce  "crutch  effects";  that  is, 
performance  breakdowns  in  combat  and  other  situations  in  which  the 
feedback  used  in  training  is  absent.  Lessons  learned  from  the  Na¬ 
tional  Training  Center,  for  example,  suggest  that  difficulty  in 
distinguishing  between  dead  and  live  targets  results  in  overkilling 
some  and  ignoring  others.  To  the  extent  that  this  discrimination  is 
simplified  in  training,  performance  in  more  realistic  situations  is 
likely  to  suffer.) 

4.  Computer  image  generation  was  revised  to  eliminate  unrealistic  tar¬ 
get  scenes;  flying  tanks,  for  example. 

5.  Haze  density  was  reduced  for  improved  visibility  of  previously  unde¬ 
tectable  targets. 

6.  Scoring  was  revised  to  reduce  the  proportion  of  students  getting  "A" 
for  performing  exercises,  and  to  increase  the  proportion  getting 
"B." 

The  identification  of  deficiencies  was,  in  our  view,  conducted  in  a 
logical,  systematic  manner;  and  resulted  in  many  useful  corrections. 

PROFICIENCY  IMPROVEMENTS 

Our  review  of  proficiency  improvements  with  the  Ml  U-COFT  was  accom¬ 
plished  using  information  from; 

1.  Sustainment  training  described  in  the  V/V  report. 

2.  Transition  training  described  in  the  V/V  report. 

3.  Reanalysi3  of  selected  sustainment  and  transition  training  data. 

4.  Canadian  Armor  Trophy  (CAT)  Team  training. 

5.  Briefing  data. 


Sustainment  Training 


Five  TC-gunner  pairs,  one  of  which  was  Mi-qualified,  received  an  orien¬ 
tation,  eye  examination,  and  U-COFT  warm-up,  totaling  about  one  hour.  They 
were  then  pretested  on  one  TC  and  four  TC-gunner  exercises,  each  comprising 
10  targets.  (See  Appendix  B.)  This  was  followed  by  approximately  30  hours 
of  U-COFT  training  over  a  period  of  three  weeks,  during  which  each  TC-gunner 


pair  completed  an  average  of  143  U-COFT  exercises.  Each  TC-gunner  pair  took 
a  posttest  on  completion  of  training,  which  was  identical  to  the  pretest 
taken  before  training. 

Comparisons  of  the  sustainment  group's  pretest  and  posttest  scores  are 
presented  later,  in  "Additional  Analyses"  (Tables  1,  2,  3,  and  4).  Those 
data  showed  substantial  improvements  in: 

1.  Acquisition  time. 

2.  Percents  of: 

(a)  Classification  errors. 

(b)  Multiple  target  engagements  completed. 

(c)  Targets  engaged. 

(d)  First  round  hits. 

(e)  Targets  hit. 

(f)  Targets  killed. 

(g)  Area  coverage  with  coaxial  machinegun. 


Transition  Trainint 


Five  TC-gunner  pairs,  three  of  which  were  qualified  on  tanks  other  than 
the  Ml,  and  two  of  which  were  not  qualified  on  any  tanks,  received  an  orien¬ 
tation,  eye  examination,  and  U-COFT  warm-up,  totaling  about  three  hours.  They 
were  not  pretested.  The  orientation  period  was  followed  by  approximately  40 
hours  of  U-COFT  training  over  a  period  of  three  weeks,  during  which  each 
TC-gunner  pair  completed  an  average  of  174  exercises.  The  TC-gunner  pairs 
were  then  tested  using  the  same  test  as  was  used  for  the  sustainment  pretest 
and  posttest. 


The  transition  group's  posttest  scores  are  presented  later,  in  "Addi¬ 
tional  Analyses"  (Tables  2,  3,  and  4).  The  transition  group's  posttest 
scores  were  as  good  as  or  better  than  the  sustainment  group's  on: 


1.  Acquisition  time. 


2.  Percents  of: 


(a)  Multiple  target  engagements  completed. 

(b)  Targets  engaged. 


(c)  First  round  hits 


(d)  Targets  hit. 

(e)  Targets  killed. 

(f)  Area  coverage  with  coaxial  machinegun. 

The  transition  group  did  less  well  than  the  sustainment  group  on  posttests  of 
identification  and  classification  errors.  The  absence  of  pretest  scores  for 
the  transition  group  or  of  a  no-training  control  group  made  it  impossible  to 
determine  the  amount  of  proficiency  gain  ascribable  to  transition  training. 

Estimates  of  the  transition  crews'  gains  relative  to  the  sustainment 
crews'  could  be  made  along  the  following  lines:  Assume  that  the  sustainment 
crews  were,  at  the  outset  of  U-COFT  training,  more  proficient  in  Ml  gunnery 
than  were  the  transition  crews.  Assume  further  that  performance  on  the 
U-COFT  test  reliably  predicts  gunnery  performance  on  the  Ml  tank.  Given 
these  assumptions,  the  transition  crews,  had  they  taken  the  U-COFT  pretest, 
would  have  scored  lower  than  the  sustainment  group  scored  on  the  same  test. 
Since  the  transition  crews'  exit-level  scores  were  equal  to  or  greater  than 
those  of  the  sustainment  crews' ,  the  gain  registered  by  the  transition  crews 
would  have  been  at  least  as  great  as  the  gain  registered  by  the  sustainment 
crews.  This  line  of  thinking  is,  in  our  view,  unwarranted  for  several  rea¬ 
sons: 

1.  The  number  of  crews  (five)  constituting  each  group  probably  was  too 
small  to  permit  reliable  inferences  about  differences  between  the 
groups'  perf-  ,.ance. 

2.  Only  one  of  the  five  sustainment  crews  was  Ml  qualified. 

3.  The  transition  crews'  entry-level  proficiency  with  Ml  gunnery  was 
unknown . 

4.  The  extent  to  which  scores  on  U-COFT  gunnery  tests  are  related  to 
scores  on  Ml  gunnery  tests  is  unknown:  The  assumption  that  soldiers 
who  are  more  proficient  in  Ml  gunnery  will  score  higher  on  a  device- 
mediated  test  than  will  soldiers  who  are  less  proficient  is  equivo¬ 
cal.  One  suspects  in  fact  that,  to  the  extent  that  a  device  differs 
(.within  limits)  from  its  parent  equipment,  a  device-mediated  test 
will  yield  greater  underestimates  of  masters'  skill  levels  than  of 
non-masters' . 

Additional  Analyses 

Additional  analyses  of  the  V/V  data  were  performed  in  an  effort  to  sepa¬ 
rate  performance  features  that  were  interdependent  in  the  V/V  data;  that  is, 
to  determine  exactly  how  and  why  performance  gains  occurred.  The  additional 
analyses  addressed: 

1.  Target  acquisition  times. 


2.  Identification  and  classification  errors. 


3.  Percents  of  target  engagements  completed 

4.  Target  engagement  results. 


Target  acquisition  times.  If  no  target  was  acquired  in  V/V,  no  time  was 
recorded.  Since  failing  to  acquire  is  more  likely  before  training  than  af¬ 
ter,  the  effect  of  this  scoring  technique  is  to  underestimate  acquisition 
times  on  the  pretest.  This  in  turn  would  yield  underestimates  of  acquisi¬ 
tion-time  gains.  Our  reanalysis  therefore  assigned  40  seconds  as  a  minimum 
estimate  of  the  missing  times  when  no  target  was  acquired  or  fired  on,  and 
estimated  the  effects  of  including  those  values  in  the  acquisition-time  aver¬ 
ages.  (Forty  seconds  is  the  maximum  target  exposure  time  for  crew  exercises 
in  the  Army's  Field  Manual  for  Ml  tank  gunnery.)  The  results  of  reanalyzing 
acquisition  times  are  shown  in  Table  1. 

Table  1 

Sustainment  Group's  Original  and  Adjusted  Mean  Acquisition  Times  (Seconds) 


for  Stationary  and 

Moving  Tank  Targets 

Engaged 

during  Day 

and  Night 

Target 

Pretest 

Posttest 

Gain 

Condition 

Or  ig 

Adj 

Orig 

Adj* 

Orig 

Adj 

Stationary  Tank 

Day 

14.3 

18.0 

8.8 

8.8 

5.5 

9.2 

Night 

12.1 

14.1 

7.5 

7.5 

4.6 

6.6 

Moving  Tank 

Day 

6.3 

7.9 

4. 1 

4.1 

2.2 

3.8 

Night 

8.8 

10.4 

4.5 

4.5 

4.3 

5.9 

Average 

10.  4 

12.6 

6.2 

6.2 

4.2 

6.4 

•No  change;  all 

targets  i 

groups  were  acquiree 

1  and  fired  on. 

Here  it  can  be  seen  that,  while  the  adjustments  proved  small,  the  average 
gain  in  acquisition  time  wa3  underestimated  by  at  least  one-third  (4.2  vs  6.4 

seconds) . 

Identification  and  classification  errors.  Identification  errors  occurred 
when  the  TC  called  a  terrain  feature  a  target,  for  example,  or  a  BMP  a  tank. 
Classification  errors  occurred  when  multiple  targets  were  engaged  in  an  order 
other  than  most  to  least  lethal.  While  identification  and  classification 
errors  were  recorded  separately  in  the  V/V,  their  numbers  depended  upon  the 
numbers  of  targets  acquired:  The  number  of  errors  could  be  minimized  by  not 


detecting  any  targets.  Our  analysis  therefore  examined  the  number  of  target 
groups  acquired  relative  to  the  number  available,  and  computed  errors  as 
percents  of  acquired  target  groups.  The  results  are  shown  in  Table  2,  where 

Table  2 

Numbers  of  Target  Groups  Acquired  and  Available, 
with  Errors  as  Percents  of  Acquired  Groups 

Sustainment  Transition 

Pretest  Posttest  Posttest 


Number  of  acquired  groups/ 
number  of  available  groups 


(per  crew) 

17/18 

18/18 

18/18 

Identification  Errors 

8.2 

5.6 

10.0 

Classification  Errors 

32.9 

17.8 

27.8 

it  can  be  seen  that  the  crews  in  the  sustainment  group  failed  on  average  to 
detect  only  one  target  group  in  the  pretest;  and  acquired,  as  did  the  transi¬ 
tion  group,  all  target  groups  in  the  posttest.  The  percent  of  identification 
errors  decreased  slightly  for  the  sustainment  group  (because  room  for  im¬ 
provement  was  slight),  and  percent  classification  errors  was  nearly  halved. 
The  reason  for  the  . . ansition  group's  relatively  high  error  percents  is  not 
clear.  The  error  data,  coupled  with  the  high  target  hit  scores  however, 
suggest  that  the  transition  group  was  shooting  first  and  asking  questions 
later. 

Percents  of  target  engagements  completed.  One  way  to  estimate  proficiency  on 
the  U-COFT  is  to  examine  differences  in  the  numbers  of  targets  engaged  before 
and  after  training.  Data  for  such  an  examination  were  available  from  the 
V/V,  but  had  to  be  converted  to  percents  for  comparability.  The  results  of 
doing  so  are  shown  in  Table  3,  where  it  can  be  seen  that  the  sustainment 
group  improved  considerably  in  its  ability  to  engage  two-  and  three-target 
arrays,  and  was  engaging  about  80  percent  of  the  number  of  available  targets 
at  the  end  of  training.  The  transition  group  was,  at  the  end  of  training, 
engaging  88.5  percent  of  the  number  of  available  targets  in  two-target  ar¬ 
rays,  and  95  percent  of  the  number  available  in  three-target  arrays. 

Target  engagement  results.  Table  M  summarizes  target  engagement  results  as 
percents  of  targets  available.  The  sustainment  group  showed  substantial 
gains  in  engaging,  hitting,  and  killing  targets,  with  exit  scores  on  kill  in¬ 
dicators  averaging  about  80  percent.  Ine  comparable  figure  for  the  transi¬ 
tion  group  was  about  85  percent. 


Summary  of  Additional  Analyses.  The  results  of  our  adiitional  analyses 
showed  that  practice  with  the  U-COFT  resulted  in  substantial  gains  in  the 


percents  of  targets  engaged,  hit,  and  killed.  The  gains  resulted  from  im¬ 
provements  in  acquisition  time,  engagement  time,  and  first  round  hits,  which 
in  turn  allowed  time  to  scan,  acquire,  and  engage  available  second  and  third 
targets. 

Table  3 

Target  Engagements  Completed,  as  Percents  of  Target  Groups  Available, 
for  Two-  and  Three-Target  Groups 


Sustainment  Transition 

Pretest  Posttest  Posttest 


Two-Target  Groups  (n=18) 

No  Targets  Acquired 

7.1 

0.0 

0.0 

No  Targets  Engaged 

10.0 

0.0 

0.0 

One  Target  Engaged 

51.4 

18.6 

11.5 

Two  Targets  Engaged 

31.4 

81.4 

88.5 

Three-Target  Groups  (n=4) 

No  Targets  Acquired 

0.0 

0.0 

0.0 

No  Targets  Engaged 

0.0 

0.0 

0.0 

One  Target  Engaged 

50.0 

5.0 

5.0 

Two  Targets  Engaged 

45.0 

15.0 

0.0 

Three  Targets  Engaged 

5.0 

80.0 

95.0 

Table  4 

Target  Engagement  Results  for 

TC-Gunner 

Pairs  and  TC 

Only, 

as  Percents  of  Numbers 

of 

Targets  Available 

Number  Targets 

Sustainment 

Transition 

Results  Available 

Pretest 

Posttest 

Posttest 

TC-Gunner  Test 

Engaged  Target 

40 

55.5 

91.0 

96.5 

First  Round  Hit  = 

Nbr  Eng'ts 

65.0 

82.3 

83.1 

Hit  Target 

40 

43.0 

79.5 

87.5 

Killed  Target 

37  Main  Gun 

42.7 

79.4 

88.6 

Area  Coverage 

3  Coaxial 

17.5 

37.5 

35.8 

TC  Test 

Acquired  Target 

10 

98.0 

100.0 

100.0 

Engaged  Target 

10 

94.0 

100.0 

100.0 

Targets  Hit 

10 

84.0 

100.0 

100.0 

Targets  Killed 

9  Main  Gun 

82.2 

100.0 

100.0 

Area  Coverage 

1  Coaxial 

52.5 

52.5 

57.5 

Canadian  Armor  Trophy  (CAT)  Team  Data 

Analyses  were  performed  of  IMJOFT  data  obtained  from  eight  CAT  TC-gunner 
pairs.  These  data  were  not  part  of  the  V/V  report,  but  were  generously  pro¬ 
vided  by  G.E.  in  support  of  our  review. 

The  training  and  testing  sequence  for  the  CAT  teams  was: 

1.  One  hour  orientation  and  warm-up. 

2.  One  hour  U-COFT  pretest. 

3.  Seven  hours  U-COFT  training. 

4.  One  hour  U-COFT  posttest  (identical  to  pretest). 

Statistically  significant  differences  (improvements)  between  pretest  and 
posttest  scores  were  found  on: 

1.  Acquisition  time. 

2.  First  round  fire  time. 

3.  First  target  hit  time. 

4.  Second  target  fire  time, 

5.  Second  target  !.it  time. 

6.  Number  of  coax  rounds  fired. 

Statistically  significant  differences  were  found  neither  in  percent  hits  with 
the  main  gun,  nor  in  target  coverag#  With  the  coaxial  machinegun.  The  CAT 
teams  were  proficient  at  hitting  targets  before  U-£OFT  training  began,  and 
were  only  slightly  more  so  after  it  ended:  their  high  pretest  scores  allowed 
little  room  for  Improvement* 

Briefing  Data 

The  results  of  U-COFT  V/V  were  summarized  in  a  briefing  by  G.E.  to 
LTG  W.  F.  Ulmer  on  16  January  1984.  Copies  of  VU-graphs  from  that  briefing 
were  provided  by  PH  TRADE  to  the  Under  Secretary  of  the  Army  in  February 
1985,  and  to  the  authors  of  this  review  in  May  1985.  Copies  of  the  VU-graphs 
which  summarized  the  results  of  sustainment  pretests  and  posttests,  and  of 
transition  posttests  are  presented  in  Appendixes  C  and  D.  Here  it  can  be 
seen  that  soldiers'  performance  on  nine  dimensions  was  reported: 

1.  Target  acquisition. 

2.  Reticle  aim. 


3.  Tracking/control  manipulation 


4.  Stationary  single  targets. 

5.  Stationary  multiple  targets. 

6.  Moving  single  targets. 

7.  Moving  multiple  targets. 

8.  Degraded-mode  gunnery. 

9.  Crew  coordination. 

Our  review  of  the  test  exercises  and  performance  data  uncovered  no  measures 
of  degraded-mode  conditions,  and  no  measures  of  crew  coordination.  Crew 
performance  with  stationary  and  single  moving  targets  also  was  not  tested, 
although  ability  might  conceivably  be  inferred  from  performance  against  mul¬ 
tiple  stationary  and  moving  targets.  We  called  G.E.  for  clarification,  and 
were  told  that  the  information  in  the  VU-graphs  was  based,  not  on  pretest  and 
posttest  scores,  but  on  V/V  test  administrators'  impressions  of  progress  in 
training.  The  data  in  the  VU-graphs  were  therefore  discarded  from  considera¬ 
tion.  Future  reports  of  performance  levels  and  gains  should  distinguish 
between  objectively  measured  test  data  and  other  kinds  of  information. 

ADDITIONAL  CONSIDERATIONS 

Our  review  of  the  V/V  report  led  to  two  additional  considerations: 

1.  Proficiency  retention. 

2.  Training  and  test  contents. 

Proficiency  Retention 

Proficiency  retention  was  examined  in  the  V/V  report  by  retesting  two 
TC-gunner  pairs  10  weeks  after  they  completed  sustainment  training,  and  by 
retesting  another  two  TC-gunner  pairs  3  weeks  after  they  completed  transition 
training.  The  sustainment  crews  showed  a  "definite  proficiency  loss"  (p.  39) 
after  10  weeks,  and  the  transition  crews  showed  none  after  three  weeks.  The 
findings  are  not  reliable  enough  to  support  making  decisions  about  retention 
or  retraining  intervals:  The  number  of  compared  crews  (two  per  group)  was 
small,  and  the  transition  group  practiced  for  about  a  third  more  time,  and 
performed  more  than  20  percent  more  exercises  (310  more  targets)  than  did  the 
sustainment  group.  The  question  of  whether  the  transition  group's  superior 
retention  was  due  to  the  amount  of  time  between  training  and  testing,  or  to 
amount  of  training  remains  open. 

Training  and  Test  Contents 

Review  of  the  information  sources  listed  earlier  yielded  several  sugges¬ 
tions  for  changes  in  U-COFT  training  and  testing.  Consideration  should,  for 
example,  be  given  to: 


1.  Expanding  the  content  of  TC  training  exercises  to  include  more  pre¬ 
firing  activities;  target  acquisition,  for  example,  and  making  ini¬ 
tial  main  gun  lays  for  gunners*  engagements.  Present  TC  training 
exercises  deal  mainly  with  firing. 

2.  Increasing  the  difficulty  of  TC  tests.  High  pretest  scores  in  the 
V/V  suggested  that  the  test  was  too  easy.  High  pretest  scores  pro¬ 
duce  ceiling  effects,  which  preclude  showing  differences  between  or 
within  groups  on  posttests. 

3.  Changing  the  TC  test  for  broader  coverage  of  the  TC  training  exer¬ 
cises.  Engagements  with  stabilization  and  laser  rangefinder  mal¬ 
functions,  for  example,  were  addressed  in  training  but  not  tested; 
as  were  50  caliber  engagements,  battlesight  engagements,  and  long- 
range  moving  targets.  The  point  here  is  that  U-COFT  (and  other) 
gunnery  tests  constitute  samples  of  the  population  or  domain  of 
possible  target  engagements.  Tests  of  proficiency  in  that  domain 
should  not  just  be  made  up  on  the  basis  of  intuitive  notions  about 
the  adequacy  of  sampling  or  covering  the  domain.  Procedures  exi3t 
for  developing  domain-referenced  tests,  and  should  be  followed  in 
constructing  tests  to  estimate  tank-gunnery  proficiency.  The  use  of 
domain-referenced  testing  procedures  would  result  in  more  reliable 
and  therefore  potentially  more  valid  proficiency  estimates  than 
those  obtained  in  the  U-COFT  V/V.  A  list  of  documents  which  de¬ 
scribe  procedures  for  constructing  domain-referenced  tank  gunnery 
tests  is  presented  in  Appendix  E. 

CONCLUSIONS 

Among  the  conclusions  which  can  be  drawn  from  this  review  are  that: 

1.  Deficiency  identification  was  conducted  in  a  logical,  systematic 
manner;  and  resulted  in  useful  hardware  and  software  improvements. 

2.  Thirty  hours  of  U-COFT  sustainment  training  produced  substantial 
average  gains  in  the  proficiency  of  a  test  group  comprising  five 
TC-gunner  pairs. 

3.  Forty  hours  of  U-COFT  transition  training  with  another  five  TC-gun¬ 
ner  pairs  produced  average  exit-level  scores  equal  to  or  greater 
than  those  obtained  by  the  sustainment  pairs.  Proficiency  gains  for 
the  transition  group,  while  probably  substantial,  could  not  be  esti¬ 
mated  accurately,  because  entry-level  proficiency  scores  were  not 
obtained. 

4.  Additional  analyses  of  test  data  suggested  that  proficiency  gains 
in  target-acquisition  time  on  the  U-COFT  were  underestimated  by 

at  least  one-third  in  the  V/V  report. 


5.  Additional  analyses  of  test  data  also  showed  substantial  gains  in 
percents  of  targets  acquired,  engaged,  hU,  and  killed.  The  gains 
resulted  from  improvements  in  acquisition  time,  engagement  time,  and 
first  round  hits,  which  in  turn  allowed  time  to  scan,  acquire,  and 
engage  available  second  and  third  targets. 

6.  Nine  hours  of  U-COFT  training  and  testing  produced  statistically 
significant  improvements  between  pretest  and  posttest  scores  of 
eight  Canadian  Armor  Trophy  (CAT)  TC-gunner  pairs  on  acquisition 
time,  first  round  fire  time,  first  target  hit  time,  second  target 
fire  time,  second  target  hit  time,  and  number  of  coaxial  machinegun 
rounds  fired.  Statistically  significant  differences  were  found 
neither  in  percent  main  gun  hits  nor  in  target  coverage  with  the 
coaxial  machinegun.  The  CAT  teams  were  so  proficient  at  hitting 
targets  before  U-COFT  training  began  that  there  wa3  little  room  for 
improvement. 

7.  Review  of  VU-graphs,  which  were  used  to  summarize  proficiency  before 
and  after  training  in  a  briefing  to  LTG  W.  F.  Ulmer,  revealed  that 
the  reported  performance  levels  and  gains  were  inconsistent  with  V/V 
test  data.  The  briefing  data  were  based,  not  on  pretest  and  post- 
test  scores,  but  on  V/V  test  administrators*  impressions  of  sol¬ 
diers'  progress  in  training. 

8.  Proficiency  loss  was,  according  to  the  V/V  report,  "definite"  after 
10  weeks,  and  nonexistent  after  three.  These  findings  are  not  re¬ 
liable  enough  to  be  used  for  making  decisions  about  retention  or  re¬ 
training  intervals:  The  number  of  compared  TC-gunner  pairs  was 
small  (two  each  in  the  10-  and  3-«eek  groups),  and  the  transition 
group  received  more  training  than  did  the  sustainment  group. 

9.  Tank  Commander  training  exercises  dealt  mainly  with  firing.  Consid¬ 
eration  should  be  given  to  expanding  the  content  of  TC  training  to 
include  more  prefiring  activities;  target  acquisition,  for  example, 
and  making  initial  main  gun  lays  for  gunners'  engagements. 

10.  The  TC  and  the  TC-gunner  tests  did  not  provide  adequate  sampling  of 
training  content.  NBC  gunnery,  for  example,  and  several  aspects  of 
degraded-mode  gunnery  were  practiced  but  not  tested.  Domain- 
referenced  methods,  described  in  several  ARI  reports,  should  be  used 
for  constructing  U-COFT  and  other  tank  gunnery  tests. 


APPENDIX  A 


LETTER  FROM  COL  J.  M.  BALL 
TO  ARMY  RESEARCH  INSTITUTE 


DEPARTMENT  OF  THE  ARMY 

OFFICE  OF  THE  PROJECT  MANAGER  FOR  TRAINING  DEVICES 
NAVAL  TRAINING  CENTER 
AEPtv  TO  ORLANDO.  FLORIDA  32813 

ATTENTION  Of 


AMCPM-ARD 


16  MAY  L35 


SUBJECT:  Response  to  Under  Secretary  of  the  Army 


ARI  Field  Unit-Fort  Knox 
Steele  Hall 

Fort  Knox,  KY  40121-5620 


1.  During  a  28  January  1985  visit  to  PM  TRADE,  the  Under  Secretary  requested  a 
copy  of  the  COFT  training  matrix  validation  test  results.  The  information  was 
furnished  the  Under  Secretary  through  HQ  AMC.  The  Under  Secretary  responded  on 
11  March  1985  with  several  comments,  one  of  which  follows:  "It  seems  to  me  that 
it  should  be  examined  carefully  and  In  detail  by  analysts  who  are  not  partisan 
in  the  U-COFT  program  area  for  the  lessons  to  be  derived.”  Consequently,  your 
office  was  contacted  and  arrangements  made  for  a  full-up  briefing  to  your  people 
on  the  validation/verification  effort  and  report. 

2.  Request  you  examine  the  report  in  detail  and  respond  per  Under  Secretary's 
request. 


JAMES  W.  BALL 
Colonel,  ORDC 
Project  Manager 


CREW  EXERCISES 


U-COFT  File 

Number  Description 


34611 


34621 

34631 

34641 


Stationary  Tank  -  Long  Range  Multiple  Stationary  and  Moving 
Targets  (Gunner  -  Precision  -  GPS  -  Normal  -  Commander  - 
Caliber  .50  -  Day) 


Stationary  Tpnk  -  Long  Range  Multiple  Stationary  and  Moving 
Targets  (Gunner  -  Precision  -  GPS  -  Normal  -  Night) 


Moving  Tank  -  Long  Range  Multiple  Stationary  and  Moving 
Targets  (Gunner  -  Precision  -  GPS  -  Normal  -  Day) 


Moving  Tank  -  Long  Range  Multiple  Stationary  and  Moving 
Targets  (Gunner  -  Precision  -  GPS  -  Normal  -  Night) 


TANK  COMMANDER  EXERCISE 


U-COFT  File 

Number  Description _ 

21211  Stationary  Tank  -  Short  Range  Single  Stationary  Targets 

(Commander  -  Precision  -  GPSE  -  Normal  -  Day  -  Malf:  GPCH) 


COFT  PRODUCTION  PROGRAM  sotuuno*  m  crorwt 

krr  fDL-j  oor  ttct  DDftcrnrNrv  l  FVFt  SYSTEMS  DCFARTMCHt 


COFT  PRODUCTION  PROGRAM 
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PORTS  OF  PROCEDURES  FOR 
RUCTING  TANK  GUNNERY  TESTS 
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